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ABSTRACT 

Two  approaches  in  designing  autostereoscopic  displays  capable  of  providing  collaborative  viewing  of  real  time  3D 
scenery  will  be  presented  and  discussed.  Both  techniques  provide  multiscopic  “look  around”  capabilities  and  are 
applicable  for  situation  rooms  or  mobile  command  centers.  In  particular,  we  discuss  a  prospective  use  of  these  displays 
for  interactive  visualization  of  detailed  three-dimensional  models  of  urban  areas,  and  the  specific  demands  associated 
with  managing  and  rendering  large  volumes  of  highly  detailed  information.  Latest  advances  in  scanning,  survey  and 
registration  in  urban  areas  have  provided  a  wealth  of  detailed  three-dimensional  data  and  imagery.  Recent  events  have 
shown  a  severe  need  and  demand  for  systems  capable  in  a  high-level  3D  visualization  upon  homeland  security  posed  by 
terrorist  actions  and  natural  disasters  within  urban  areas,  as  well  as  for  military  operations  in  urban  terrain  (MOUT). 
The  capacity  to  visualize  sightlines,  airflow,  flooding,  and  traffic  in  real  time  3D  within  dense  urban  environments  is 
increasingly  critical  for  military  and  civilian  authorities,  as  well  as  urban  planners  and  city  managers.  Development  of  a 
high-quality  3D  imaging  systems  is  critical  also  for  such  areas  as  medical  data  imaging,  gaming  industry,  mechanical 
design  and  rapid  prototyping. 
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1.  INTRODUCTION 

A  significant  proportion  of  the  Earth’s  population  lives  in  dense  urban  areas,  which  has  led  to  an  increasing  emphasis  on 
the  need  to  visualize  the  complex  structure  of  such  environments.  Natural  disasters,  such  as  the  Hurricane  Katrina,  the 
Southeast  Asian  tsunami  and  their  aftermath,  military  operations,  and  homeland  security  have  all  demonstrated  the 
increasing  need  for  decision  making  tools  allowing  civilian  and  military  authorities  to  both  plan  and  monitor  operations 
within  urban  areas.  In  particular,  three-dimensional  visualizations  of  dense  urban  areas  are  desirable  in  that  they  not 
only  constitute  three  dimensional  surfaces,  but  also  the  interior  volume  of  various  facilities,  such  buildings, 
underground  bunkers,  tunnels,  utilities  and  infrastructure.  With  the  advent  of  such  programs  as  Google  Earth,  the  two 
dimensional  visualization  of  three-dimensional  scenes  has  become  increasingly  common,  but  many  of  the  practical  uses 
of  the  technology  are  complicated  by  the  ambiguities  of  two-dimensional  perspectives  of  three-dimensional  scenes.  In 
particular,  gauging  the  three  dimensional  position  of  paths  and  points  as  well  as  dealing  with  multlayered  translucency. 
As  a  result  the  development  of  true  3D  displays  allowing  for  collaborative  viewing  of  urban  scenes  will  allow  for  a 
denser  yet  more  intuitive  visualization  of  urban  areas. 

Within  this  paper,  we  present  the  architecture  of  two  types  of  autostereoscopic  displays  that  are  well  suited  to  the 
collaborative  display  of  urban  environments.  We  will  also  consider  corresponding  software  that  enables  the  display 
urban  areas  derived  from  actual  data  or  from  architectural  models.  Both  display  types[l,2]  are  autostereoscopic  and 
multiscopic  [3]  in  that  they  do  not  require  glasses  or  goggles  for  viewing  and  simultaneously  displaying  multiple 
perspectives  allowing  for  “look  around”  viewing  of  scenes  with  multiple  viewers.  This  is  achieved  by  exploiting  the 
very  high  framerate  possible  with  deformable  mirror  devices  (DMDs)  to  implement  temporal  multiplexing  [4].  In  the 
second  design  the  imaging  geometry  is  coupled  with  the  use  of  the  aspect  from  points  (AFP)  approach  [5,6]  by  applying 
holographic  optical  elements  to  redirect  the  multiplexed  images.  The  operation  is  enhanced  through  advances  in 
graphical  processing  unit  (GPU)  technology  coupled  with  improvements  in  computer  bus  technology  to  allow  for  the 
real  time  generation  of  the  necessary  perspectives  and  the  high  bandwidth  transfer  of  the  corresponding  information  to 
the  DMD  unit. 
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In  addition  we  describe  the  integration  of  urban  models  based  on  the  work  of  one  of  the  authors  [8,9,10]  within  the 
display  system.  Due  to  the  proportionally  greater  rendering  requirements  associated  with  the  multiple  perspectives  it  is 
essential  to  optimize  rendering  times.  As  a  result  it  is  highly  desirable  to  incorporate  a  combination  of  level  of  detail 
(LOD)  data  in  conjunction  with  scenegraph-based  modeling  to  allow  for  optimal  control  of  rendering  complexity.  Thus 
the  virtual  reality  modeling  language  (VRML)  representations  generated  by  the  scanning  and  fusion  software  are 
interpreted  by  a  customized  rendering  engine  to  allow  for  generation  of  the  appropriate  oblique  perspectives  with 
optimal  performance. 


2.  DISPLAY  HARDWARE 

The  capacity  to  display  rendered  computer  models  in  true  3D  has  existed  for  a  considerable  amount  of  time,  but  until 
recently,  viewing  of  the  imagery  required  special  glasses  or  goggles.  While  such  methods  have  seen  wide  adoption  in 
oil  and  gas  exploration  and  scientific  visualization  where  such  tools  are  essential,  widespread  use  has  been  limited  due 
to  the  cumbersome  nature  of  the  viewing  equipment.  With  the  recent  advent  of  practical  autostereoscopic  displays  it  is 
possible  to  display  stereoscopic  scenes  without  the  need  for  glasses  or  goggles.  The  first  generation  of  autostereoscopic 
displays  have  primarily  generated  stereoscopic  views  suited  to  individual  viewers  with  two  viewing  zones  and  limited 
look  around  capabilities.  New  designs  allow  for  multiple  perspectives  distributed  across  multiple  viewing  zones 
allowing  for  both  look  around  and  multiple  viewers.  Both  types  of  the  true  3D  displays  we  employ  for  urban 
visualization  incorporate  multiscopic  capabilities  allowing  for  collaborative  evaluation  of  the  imagery.  The  two 
configurations  described  in  this  paper  employ  temporal  multiplexing  as  opposed  to  the  more  common  spatial 
multiplexing  (lenticular  and  parallax  barrier)  techniques.  The  first  display  utilizes  conventional  optics  while  the  second 
also  incorporates  a  holographic  optical  element  to  allow  for  more  flexible  redirection. 

2.1.  Temporally  Multiplexed  Autostereoscopic  Display  (TMAD) 

The  design  of  the  TMAD  is  based  on  the  conventional  optics,  and  its  primary  components  are  shown  in  Figure  1.  A 
high  speed  DMD  based  SLM  is  imaged  onto  a  diffuser  element  via  a  relay  lens  forming  a  real  image.  This  image  is 
then  imaged  by  a  second  two-element  lens  system  (consisting  of  the  front  and  rear  projection  lens  assemblies),  which 
relays  the  image  from  the  diffuser  to  a  Fresnel  field  lens.  Interspersed  between  two  elements  of  the  secondary  lens 
system  at  the  pupil  is  a  high  speed  represented  by  the  intervening  box  shutter,  which  is  synchronized  with  the  projection 
engine  to  generate  a  slit  corresponding  to  the  perspective  being  displayed  on  the  projection  engine.  This  slit  is  imaged  to 
the  exit  aperture,  which  corresponds  to  the  viewing  position.  Each  position  of  the  slit/exit  aperture  has  a  matched  image 
associated  with  it.  By  projecting  images  and  operating  the  shutter  at  a  rate  sufficient  to  achieve  persistence  of  vision  all 
of  the  viewing  zones  remain  visible  at  the  same  time,  creating  a  multiscopic  display. 


Figure  1.  Temporally  Multiplexed  Autostereoscopic  Display. 

This  approach  has  the  advantage  of  generating  a  large  (8-10)  number  of  viewing  zones  through  a  single  optical  train 
wile  retaining  the  full  resolution  of  the  projection  engine.  The  primary  challenge  associated  with  TMAD  architecture  is 
due  to  an  exceedingly  high  frame-rate  required  to  simultaneously  project  the  multiple  images  requisite  for  the  3D 
operation.  This  requires  the  use  of  a  projection  engine,  which  incorporates  a  correspondingly  high  frame-rate  spatial 
light  modulator,  as  well  as  the  necessary  rendering  hardware  and  high  bandwidth  connection  transfer  the  rendered 
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frames.  Advances  in  the  raw  frame-rates  of  DMD  based  SLMs  coupled  with  advances  in  the  speed  of  both  modern 
GPUs  and  their  associated  data  busses  have  greatly  increased  the  viability  of  time  multiplexed  designs. 

The  initial  prototype  of  the  TMAD  shown  in  Figure  2  has  demonstrated  up  to  six  viewing  zones  in  conjunction  with  up 
to  four  independent  perspectives.  The  primary  limitation  of  the  initial  system  was  the  use  of  a  conventional  projector  to 
provide  imagery.  As  a  result  a  maximum  framerate  of  85  frames  per  second  was  available,  limiting  the  number  of 
independent  perspectives  to  4  before  the  level  of  image  flicker  became  intolerable.  The  prototype  system  has  been 
demonstrated  in  conjunction  with  both  multiscopic  imagery  and  stereoscopic  imagery,  allowing  for  use  of  conventional 
3D  programming  as  well  as  pre-rendered  multi -perspective  images. 


Figure  2.  Initial  Prototype  of  the  Time  Multiplex  Autostereoscopic  Display. 

Design  of  the  follow-on  system  is  currently  underway.  The  new  system  employs  the  use  of  a  Texas  Instruments  DMD 
Discovery  3000  SLM  to  provide  extremely  rapid  display  of  frames  (  480  fps)  coupled  with  the  use  of  either  a  digital 
visual  interface-digital  (DVI-D)  or  low  voltage  differential  signaling  (LVDS)  coupled  with  the  PCI-express  (PCI-e)  bus 
to  achieve  multiple  gigabit  per  second  link  between  one  or  more  GPUs  and  the  DMD  buffer  system.  Given  the  high 
framerates  required,  the  use  of  a  combined  color  wheel  and  metal  halide  lamp  which  is  commonly  employed  with 
conventional  DMD  projectors  is  not  desirable  due  to  the  high  speed  of  operation  required  for  the  color  wheel  as  well  the 
challenges  associated  with  synchronizing  the  variations  of  rotation  speed.  As  a  result  the  integration  of  a  LED  based 
light  engine  is  under  development.  LED  Light  engines[7]  for  DMD  projectors  have  the  advantage  of  very  rapid 
switching  times  (~1 00ns),  highly  saturated  color  gamuts  and  relatively  high  levels  of  illumination.  The  design 
parameters  of  the  system  assume  a  20”  diagonal  screen  with  8-16  viewing  zones.  The  ultimate  aim  of  the  system  is  to 
integrate  the  display  under  development  as  module  of  a  clustered  display  which  employ  each  individual  display  as  a 
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“pie  slice”  of  multiple  displays  which  may  be  combined  to  form  a  synchronously  networked  display  as  shown  in  Figure 
3.  Such  an  arrangement  allows  for  the  use  of  cluster  rendering  and  multiple  optical  trains  to  allow  for  broad  fields  of 
view  well  suited  to  urban  visualization. 


Figure  3.  “Pie  Slice”  Display  Cluster. 


2.2  Aspect  From  Point  Display  (AFPD) 

The  AFPD  consists  of  a  time  multiplexed  display  system  which  incorporates  both  conventional  optics  and  a  faceted 
holographic  optical  element  (FlOE)  allowing  for  greater  flexibility  in  the  optical  design.  Two  configurations  are  possible 
using  this  approach.  The  first  integrates  a  series  of  holographic  facets  near  the  pupil  of  the  system  which  image  to  the 
exit  aperture  in  a  manner  similar  to  that  of  a  conventional  time  multiplexed  display.  The  second  configuration  consists 
of  a  series  of  facets  near  the  image  plane  of  the  system,  which  selectively  displays  portions  of  separate  images  projected 
by  the  high-speed  projector  system.  This  latter  approach  is  similar  to  that  of  integral  display  systems.  Figure  4  illustrates 
this  configuration.  The  AFPD  is  still  in  the  early  stages  of  development. 
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Figure  4.  Aspect  From  Points  Display  Concept. 
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3.  MODEL  CAPTURE  AND  IMPORT 

In  order  to  present  timely  and  accurate  models  of  urban  landscapes  it  is  necessary  to  be  able  to  capture  as  well  as  import 
such  3D  models.  We  have  employed  both  methods  to  visualize  urban  landscapes,  incorporating  data  collected  by  one  of 
the  authors  [8,9,10]  as  well  as  data  derived  from  sources  such  as  Google  Earth. 

3.1.  Model  Capture 

The  rapid  and  efficient  capture  of  3D  models  of  urban  areas  is  of  great  interest  for  several  reasons.  With  increased 
storage  capacity  it  is  possible  to  store  3d  models  of  major  urban  areas  for  the  purposes  of  navigation  planning  and 
visualization,  as  a  result  a  variety  of  businesses  most  notably  Google  are  pursuing  the  collection  of  such  information. 
Given  the  sheer  volume  of  data  required  for  collection,  rapid,  largely  automated  methods  will  be  required.  In  the  case  of 
military  command  and  control  as  well  as  civilian  response  to  disasters  or  terrorist  attacks,  the  need  exists  to  map  new 
areas  as  well  as  obtain  information  on  damage  inflicted  by  battle  or  nature.  In  both  cases,  a  means  of  rapidly  capturing 
the  3D  structure  of  an  urban  area  is  required.  In  addition  it  is  equally  essential  to  georeference  these  models  in  order  to 
allow  fusion  with  existing  maps  as  well  as  position  data  generated  by  GPS  tracking  of  vehicles  and  personnel. 

The  primary  method  of  capture  employed  in  our  displays  utilizes  a  combination  of  aerial  photography  and  LIDAR  scans 
to  obtain  the  upper  portions  of  buildings  and  structures  as  well  as  templates  for  registration  of  the  models,  coupled  with 
ground  level  scans  to  obtain  the  facades  of  buildings  and  structures.  The  sequence  of  the  algorithm  is  as  follows: 

•  Aerial  Photography  and  Registered  Lidar  scanning  is  performed  at  medium  altitude.  Point  clouds  of  contiguous 
upper  portions  of  buildings  and  structures  are  polygonized  via  Random  Sample  Consensus  (RANSAC)-like 
algorithm  forming  the  upper  portion  of  the  3D  model. 

•  The  outlines  of  the  lower  portions  of  the  model  are  formed  by  the  outlines  of  the  upper  model. 

•  Vehicles  equipped  with  registered  cameras  and  two  laser  scanners  then  crisscross  the  streets  of  the  area  of 
interest.  One  laser  scanner  collects  a  vertical  profile  which  is  coupled  with  the  registered  camera  to  obtain  the 
actual  structure  of  the  facades  while  a  second  horizontal  laser  scanner  estimates  the  position  of  the  vehicle 
based  on  the  fa9ade  outline  coupled  with  dead  reckoning  (it  is  assumed  that  GPS  coverage  is  sporadic  or 
distorted  by  clutter  and  multipath  interference). 

•  The  point  cloud  model  is  then  registered  to  the  faqade  outline  via  the  use  of  a  by  means  of  Monte-Carlo- 
Localization  (MCL),  a  powerful  Bayesian  particle  filtering  algorithm.  Finally  the  lower  point  cloud  is  surfaced 
to  form  a  contiguous  3D  model  of  the  scene. 

The  resulting  raw  model  is  then  translated  into  a  scenegraph  format  (VRML97)  incorporating  multiple  level  of  detail 
(LOD)  models  in  order  to  allow  for  the  precise  control  of  image  framerates.  This  is  of  particular  importance  in  the  case 
of  time  multiplexed  display  systems  in  that  is  necessary  to  render  each  frame  for  each  of  the  perspectives  within  the 
refresh  rate  of  the  displays. 

The  Primary  model  used  in  out  present  work  is  shown  in  Figure  5  and  consists  of  a  model  of  downtown  Berkley 
California  generated  by  the  method  described  above.  The  final  model  has  been  optimized  for  LOD  control  to  optimize 
render  times.  This  model  has  been  successfully  rendered  for  3D  visualization  employing  both  stereoscopic  and 
multiscopic  techniques,  which  will  be  described  in  the  following  section. 
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Figure  5.  Scanned  Model  of  Urban  Area. 


3.2.  Model  Import 

While  the  highly  detailed  models  obtained  by  direct  scanning  of  urban  areas  provide  an  optimal  source  of  models  for 
specific  areas  of  interest,  it  is  often  necessary  to  incorporate  less  detailed  models  covering  a  broader  overall  area.  A 
good  example  of  such  a  set  of  low  detail  models  is  the  set  of  models  employed  in  Google  Earth  for  their  3D 
representations  of  large  urban  areas.  Despite  the  lack  of  detail  of  these  models  they  are  often  sufficient  to  assess 
important  features  of  urban  terrain  such  as  lines  of  sight,  lines  of  flight,  and  airflow.  Thus  the  capacity  to  import  this 
information  is  often  essential.  Given  that  a  broad  variety  of  differing  formats  and  methods  for  access  are  employed  for 
differing  data  sets.  In  order  to  avoid  the  complexity  of  such  protocols  our  initial  proof  of  concept  approach  employed 
intercepting  the  urban  model  at  the  level  of  the  scene  description  language.  In  particular  models  were  extracted  by 
employing  the  OpenGL  Extractor  (OGLE)  [11]  package  to  capture  the  three  dimensional  scene  by  capture  of  the  stream 
of  incoming  OpenGL  commands.  The  sequence  was  then  parsed,  filtered  and  then  reregistered  to  obtain  a  model 
compatible  with  our  software  an  example  of  which  is  shown  in  Figure  6.  Future  refinements  of  this  process  will  operate 
via  direct  access  of  the  selected  3D  databases. 


Figure  6.  Image  of  Manhattan  Island  color  coded  by  building  height . 
(Source  Data  ©  2006  Google,  ©  2006  Sandborn,  ©  2006  Europa  Technologies) 


6 


4.  SOFTWARE  METHODS 


Visualization  of  multiscopic  computer  graphics  for  specific  3D  display  configurations  imposes  unique  demands  upon 
the  rendering  software  employed.  For  each  distinct  viewing  zone  an  image  with  the  appropriate  shear  perspective 
matched  to  the  geometry  of  the  viewer  and  the  display  must  be  generated  [12,13,14].  This  rendering  process  must  be 
performed  at  a  high  rate  of  speed  to  allow  for  smooth  interaction  with  the  model.  In  addition,  given  n  views  the 
rendering  system  must  maintain  2n  buffers  to  allow  for  proper  display  (a  practice  analogous  to  double  buffering  in 
monoscopic  rendering  and  quad  buffering  in  stereoscopic  rendering).  As  a  result  it  is  generally  necessary  to  extend  the 
rendering  engine  employed  to  support  these  requirements. 

Within  our  present  implementation  of  the  urban  visualization  system  several  rendering  engines  have  been  employed  to 
achieve  stereoscopic  and  multiscopic  viewing  of  urban  scenes.  The  first  approach  is  to  employ  conventional 
stereoscopic  rendering  engines  and  alternate  between  left  and  right  views.  This  results  in  a  series  of  viewing  positions 
that  alternate  between  stereoscopic  and  pseudoscopic  (i.e.  inside  out)  imagery.  These  stereoscopic  approaches  may  be 
achieved  with  integrated  stereoscopic  drivers  such  as  the  consumer  stereo  driver  provided  by  nVidia.  This  allows  for  the 
direct  use  of  many  preexisting  rendering  engines  and  VRML  browsers  without  appreciable  modification.  This  has 
allowed  for  the  display  of  our  urban  models  via  several  different  rendering  engines,  including  in-house  display 
programs  as  well  as  the  Cortona  VRML  brouser  developed  by  Parallel  Graphics. 

While  the  use  of  the  stereoscopic  display  mode  allows  for  simple  compatibility  with  preexisting  software,  the  presence 
of  pseudoscopic  viewing  zones  as  well  as  lack  of  look  around  capabilities  limits  its  utility.  As  a  result  it  is  necessary  to 
develop  true  multiscopic  display  software.  Initial  development  of  multiscopic  displays  focused  on  the  use  of  pre¬ 
rendered  animations  of  display  sequences  in  conjunction  with  synchronized  playback  displays.  Within  our  initial  display 
system  the  Guimigolus  extension  of  the  VisionEgg  vision  stimulation  system15.  This  allows  for  stable  display  of 
multiple  viewing  zones  (4-6)  but  is  limited  to  the  framerates  of  conventional  displays  (<140  fps). 

In  order  to  allow  for  the  simultaneous  display  of  8-16  viewing  zones  we  are  presently  developing  a  custom  rendering 
engine  which  is  optimized  for  rapid  rendering  of  imagery  with  moderate  complexity  and  dynamic  range  while  allowing 
for  direct  transfer  of  the  imagery  to  the  DMD  based  SLM  employed  by  our  displays.  In  order  to  achieve  this,  our  present 
focus  is  on  the  use  of  LOD  control  to  strictly  limit  rendering  time,  the  use  of  multiple  GPU  arrangements  via  the 
scalable  link  interface  (SLI)  architecture  and  the  development  of  rapid  transfer  methods  between  the  graphics  buffer 
memory  and  the  DMD  buffer. 


5.  CONCLUSION 

The  visualization  of  dense  urban  areas  is  of  increasing  importance  for  military,  and  civilian  applications.  The 
complexity  of  such  visualizations  introduces  ambiguities  that  are  difficult  to  resolve  within  conventional  displays.  The 
use  of  true  3D  displays  which  incorporate  multiscopic  “look  around”  and  collaborative  capabilities  resolve  many  of 
these  limitations.  Within  this  paper  we  have  described  the  development  of  multiscopic  displays  that  have  been  used 
successfully  to  visualize  models  derived  from  direct  scanning  as  well  as  preexisting  models  of  urban  areas. 
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